Skip to content

feat: adding the operation delete_all_documents to the OpenSearchDocumentStore #2321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Oct 7, 2025
Merged

feat: adding the operation delete_all_documents to the OpenSearchDocumentStore #2321

merged 12 commits into from
Oct 7, 2025

Conversation

@davidsbatista
Copy link
Contributor

@davidsbatista davidsbatista commented Sep 25, 2025 •
edited by sjrl
Loading

Related Issues

Proposed Changes:

  • adds a delete_all_documents() method to the OpenSearchDocumentStore class to both Synchronous and Asynchronous versions
  • recreate_index=False (default): Uses delete_by_query API for faster deletion of large datasets
  • recreate_index=True: Recreates the entire index, preserving original mappings and settings
  • other small improvments:
    • _deserialize_search_hits() → @staticmethod
    • _process_bulk_write_errors() → @staticmethod
    • _deserialize_document() → @staticmethod
    • _postprocess_bm25_search_results() -> @staticmethod
  • minor typo corrections in docstrings
  • added proper SPDX license headers to test files

📝 Usage Example

# Delete all documents (fast, keeps index structure)
document_store.delete_all_documents()

# Delete all documents and recreate index (slower, resets index state)
document_store.delete_all_documents(recreate_index=True)

How did you test it?

  • unit tests, integration tests + CI tests
  • test_document_store.py - Added test_delete_all_documents() with and without index recreation
  • test_document_store_async.py - Added test_delete_all_documents_async() with and without index recreation

Checklist

@github-actions github-actions bot added integration:opensearch type:documentation Improvements or additions to documentation labels Sep 25, 2025
@davidsbatista davidsbatista changed the title Feat/adding delete all docs to open search document store feat: adding the operation delete_all_documents to the OpenSearchDocumentStore Sep 25, 2025
@davidsbatista davidsbatista marked this pull request as ready for review September 25, 2025 14:15
@davidsbatista davidsbatista requested a review from a team as a code owner September 25, 2025 14:15
@davidsbatista davidsbatista requested review from julian-risch and removed request for a team September 25, 2025 14:15
@anakin87
Copy link
Member

anakin87 commented Sep 26, 2025

Just a quick thought... I am not sure deleting and recreating the index is a good idea.

As a user, I'd expect delete_all_documents to only remove the data, not touch the index itself. And even though we recommend managing the DB via Haystack, some users might still have changed index attributes manually. In that case, recreating the index could cause surprises.

A bulk delete might be a safer option. Maybe it's also worth checking how this was done in Haystack 1.x.

@davidsbatista
Copy link
Contributor Author

davidsbatista commented Sep 26, 2025 •
edited
Loading

I was investigating into it and for both Elastic and Open, specially for large volumes of data, this is the most efficient way to do it. In both cases the indexes are recreated with the same mappings/settings.

The only issue is if a user changes the index after creation/population.

@davidsbatista
Copy link
Contributor Author

davidsbatista commented Sep 26, 2025

What about something like:

def delete_all_documents(self, recreate_index=False)

By default we apply a usual delete operation, and if recreate_index=True, we just recreate the index using the mappings/settings defined by the user or the default ones.

@anakin87
Copy link
Member

anakin87 commented Sep 26, 2025

What about something like:

def delete_all_documents(self, recreate_index=False)

I would prefer this idea!

julian-risch
julian-risch requested changes Sep 26, 2025
View reviewed changes
Copy link
Member

@julian-risch julian-risch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks quite good to me already. My main suggestion is that if we have a try except in delete_all_documents_async, it makes sense to add the same to the sync implementation of delete_all_documents to ensures consistent behavior.
Other than that, I have a suggestion for the tests to make sure they only test one thing. You could create a separate test, for example test_index_functionality_after_delete_all_documents for testing writing and retrieving of documents after deleting all documents. Something like:

def test_index_functionality_after_delete_all_documents(self, document_store):
    """Test that documents can be written and retrieved after delete_all_documents"""
    document_store.write_documents([Document(id="1", content="Test")])  # or add more documents
    document_store.delete_all_documents()
    
    new_doc = Document(id="2", content="New test")
    document_store.write_documents([new_doc])  # or add more documents
    assert document_store.count_documents() == 1
    
    results = document_store.filter_documents()
    assert len(results) == 1
    assert results[0].content == "New test"

@julian-risch julian-risch mentioned this pull request Sep 29, 2025
Open
@phigep phigep mentioned this pull request Sep 30, 2025
Merged
julian-risch
julian-risch approved these changes Oct 7, 2025
View reviewed changes
Copy link
Member

@julian-risch julian-risch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! 👍 _prepare_delete_all_request in particular is a good alternative to recreating the index and the default of delete_all_documents(recreate_index=False) makes sense to me.

@davidsbatista davidsbatista merged commit f7c797f into main Oct 7, 2025
6 checks passed
@davidsbatista davidsbatista deleted the feat/adding-delete-all-docs-to-OpenSearchDocumentStore branch October 7, 2025 12:28
This was referenced Oct 18, 2025
Closed
Merged
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@julian-risch julian-risch julian-risch approved these changes

Assignees

No one assigned

Labels

integration:opensearch type:documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

add delete_all_documents() operation to OpenSearchDocumentStore

4 participants

@davidsbatista @anakin87 @julian-risch